Online Multi-Task Gradient Temporal-Difference Learning

Reinforcement learning (RL) is an essential tool in designing autonomous systems, yet RL agents often require extensive experience to achieve optimal behavior. This problem is compounded when an RL agent is required to learn policies for different tasks within the same environment or across multiple environments. In such situations, learning task models jointly rather than independently can significantly improve model performance (Wilson et al. 2007; Lazaric and Ghavamzadeh 2010). While this approach, known as multi-task learning (MTL), has achieved impressive performance, the performance comes at a high computational cost when learning new tasks or when updating previously learned models. Recent work (Ruvolo and Eaton 2013) in the supervised setting has shown that online MTL can achieve nearly identical performance to batch MTL with large computational speedups. Building upon this approach, which is known as the Efficient Lifelong Learning Algorithm (ELLA), we develop an online MTL formulation of model-based gradient temporal-difference (GTD) reinforcement learning (Sutton, Szepesvári, and Maei 2008). We call the proposed algorithm GTD-ELLA. Our approach enables an autonomous RL agent to accumulate knowledge over its lifetime and efficiently share this knowledge between tasks to accelerate learning. Rather than learning a policy for an RL task tabula rasa, as in standard GTD, our approach rapidly learns a high performance policy by building upon the agent’s previously learned knowledge. Our preliminary results on controlling different mountain car tasks demonstrates that GTD-ELLA significantly improves learning over standard GTD RL.